Skip to content

docs(contributing): tell contributors how to produce evidence, not just tests - #706

Merged
ffroliva merged 1 commit into
developfrom
docs/contributing-spike-evidence
Sep 6, 2026
Merged

ffroliva merged 1 commit into
developfrom
docs/contributing-spike-evidence

Conversation

@ffroliva

@ffroliva ffroliva commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Summary

CONTRIBUTING.md told contributors how to prove a fix (the E2E evidence layer, #675) and nothing about how to prove a diagnosis — which is what a bug report is actually made of. The spike harnesses became first-class for agents in #704; this makes them findable by people.

Adds one section under the existing evidence table:

  • the two harnesses — in-process scripts/dev/spike_*.py vs the hand-driven CDP/HAR capture in scripts/dev/har-spike/ — and which to reach for
  • the rule: a selector that does not match is evidence about the selector, never about the feature. A timeout tells us an anchor missed, not that the thing is gone. Saying "X is not supported on this host" without a DOM or network observation is how a working feature got documented as impossible here, and it cost a day to undo (fix: drive character create on the migrated Flow host #703).
  • redaction, explicitly — this is the one place we invite someone to attach a capture. Raw HARs carry cookies, Bearer tokens and prompts, so attach extract_har_summary.py's redacted summary, never the raw file. *.har is gitignored repo-wide for the same reason.

The limitation, stated rather than discovered

The CDP harness is nine PowerShell scripts and Windows-only today (launch-flow-chrome.ps1 alone hardcodes four Windows paths). Rather than let a macOS or Linux contributor find that out the hard way, the section says so and points them at the in-process probes or an incident bundle.

A portable Python twin is worth doing — Playwright's channel="chrome" and _spike_common.resolve_profile_dir() already solve both Windows-bound problems, and probe_agent_mode.py is the existing pattern — but incrementally, one script at a time when a spike needs it, not as a 1691-line rewrite nobody can review. Not in this PR.

Test plan

  • check_doc_links.py — all links resolved across 29 files (caught nothing, but the
    docs/DEBUGGING.md#automatic-incident-bundles anchor is real)
  • check_repo_hygiene.py — 1012 tracked files, no violations
  • 10 documentation-gate tests pass
  • Verified rather than guessed: my first draft invented ~/.local/share/gflow-cli/incidents/
    as the bundle path. The canonical form is <GFLOW_CLI_HOME>/incidents/, per
    docs/CONFIGURATION.md and docs/DEBUGGING.md. Corrected before commit.
  • Docs-only, no Flow surface touched, so no e2e applies

…st tests

CONTRIBUTING had an "E2E is the evidence layer" rule and no mention of the
spike harnesses, so an outside contributor learned how to prove a FIX and
nothing about how to prove a DIAGNOSIS. That is the half a bug report is
actually made of.

Adds a short section under the same table: the two harnesses (in-process
`spike_*.py` and the hand-driven CDP/HAR capture), when each applies, and
the rule they exist to enforce —

    A selector that does not match is evidence about the selector.
    It is never evidence about the feature.

Also states the redaction path explicitly, because this is the one place a
contributor is invited to attach a capture: raw HARs carry cookies, Bearer
tokens and prompts, so attach `extract_har_summary.py`'s redacted output,
never the raw file.

Records the platform limitation honestly rather than letting someone
discover it: the CDP harness is PowerShell and Windows-only today, so macOS
and Linux contributors are pointed at the in-process probes or an incident
bundle instead. A portable Python twin is being ported one script at a
time, with `probe_agent_mode.py` as the pattern.
@ffroliva
ffroliva merged commit 7b01711 into develop Sep 6, 2026
15 checks passed
@ffroliva
ffroliva deleted the docs/contributing-spike-evidence branch October 22, 2026 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant